This disclosure relates generally to data management. More specifically, the disclosure provided herein relates to data feed management.
Data feed management has become important for entities relying upon and/or offering data applications. These data applications often depend upon reliable data delivery to support data collection, data correlation, and analysis of the collected and/or correlated data. The data consumed by these data applications may be collected from a wide variety of sources and organizations, each of which may use varied mechanisms for tracking and/or delivering the data to the consumers or subscribers. For example, some data are streamed in real time to the data applications, while other data are obtained at intervals or collected in an on-demand or ad hoc manner by the data applications.
Because of these varied mechanisms for tracking and delivering data, individual data applications may make individually-tailored arrangements with data providers to subscribe to data feeds or data sources. Thus, subscribers often are not aware of a structure of incoming data streams or data files. As such, the subscribers often must determine or discover the structure of the data streams or data files. Discovery of the file structure can be a difficult process, and may need to be completed by each subscriber that joins or subscribes to a feed.
Some organizations generate the data feeds and subscribe to the data feeds as part of their internal operations. For example, a network operator may generate a wide variety of continuous streams of data files from operations at geographically distributed locations, and may stream these data to a monitor system for correlation and analysis for understanding and controlling its operations.
Because the data sources may be geographically dispersed and may be sporadically connected to a network, collection of these data in an efficient and reliable manner can be difficult. Furthermore, some or all of the data streams may be relevant to some analysts in one geographic location, while other data streams may be relevant to other analysts in another geographic location. Delivering the relevant data feeds to the relevant entities for analysis is a tedious, time-consuming, and difficult task.